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1. 


EXECUTIVE  SUMMARY 


This  research  addressed  numerous  technological  challenges  in  developing  the 
necessary  methodologies  and  tools  for  a  speedup  of  the  core  services  of  the  Reference 
Implementation  of  the  Joint  Battlespace  Infosphere  (JBI).  In  the  current  world 
environment,  the  amount  of  available  information  has  dramatically  increased, 
compounding  the  problem  of  providing  the  right  information  to  the  right  people  in  a 
timely  and  accessible  fashion.  Potential  adversaries  have  access  to  much  of  the  same 
information  that  we  do.  This  dynamic  world  environment  has  established  the  necessity 
that  we  surpass  our  adversaries  in  the  timeliness  that  information  resources  can  be 
leveraged.  This  research  investigated  techniques  that  accelerated  and  scaled  the  baseline 
JBI  by  100  times  through  the  use  of  high  perfonnance  computers,  parallel  programming 
techniques  and  optimizations. 


It  was  the  goal  that  the  100  X  JBI  optimized  the  performance  of  publish  and 
subscribe  services  that  are  required  for,  as  an  example,  an  Air  Operations  Center  (AOC). 
Such  a  system  will  allow  either  a  100  fold  increase  in  the  amount  of  information  to  be 
processed  by  the  JBI  core  services  in  the  same  amount  of  time,  a  100  fold  decrease  in  the 
amount  of  time  necessary  to  process  the  same  amount  of  core  information,  or  some 
weighted  combination  of  both.  The  purpose  of  this  system  was  to  accelerate  the  core 
services  of  the  Joint  Battlespace  Infosphere  by  two  orders  of  magnitude  so  that  the  JBI  is 
scaled  for  use  in  a  warfighting  environment.  The  JBI  is  an  information  management 
system  which  can  be  tailored  to  provide  the  right  information  to  the  warfighter  at  the 
right  time,  without  at  the  same  time  overwhelming  the  warfighter  with  extraneous 
information. 

The  system  was  tested  utilizing  the  Defense  Research  and  Engineering  Network 
(DREN),  over  which  remote  high  perfonnance  computers  were  accessed  in  an  interactive 
mode.  Two  local  high  performance  computers,  Coyote  and  the  Heterogeneous  High 
Performance  Computer  (HHPC),  were  accessed,  as  well  as  Seahawk  at  Space  and  Naval 
Warfare  Systems  Center,  Sandiego  (SSCSD),  Powell  at  the  Army  Research  Laboratory 
(ARL),  Mach  2  at  ASC  WPAFB,  and  the  Koa  Cluster  at  the  Maui  High  Performance 
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Computing  Center  (MHPCC).  The  accelerated  core  services  of  the  JBI  were 
demonstrated  within  this  distributed  environment. 

This  two  year  report  focuses  on  the  speedups  of  the  core  services  of  the  100X  JBI. 
In  the  final  report,  applications  of  this  architecture  will  also  be  presented. 
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2. 


JBI  OVERVIEW 


An  overview  of  the  Joint  Battlespace  Infosphere  (JBI)  is  presented  in  this  section. 
2.1  Introduction 

The  JBI  is  an  information  management  system  based  on  the  publish/subscribe 
distributed  systems  paradigm.  The  JBI  provides  a  means  for  a  Community  of  Interest 
(COI)  to  share  information  in  a  centralized  and  organized  fashion.  JBI  clients  are 
separated  into  two  main  groups,  the  publishers  (producers  of  information)  and  the 
subscribers  (consumers  of  information).  The  users  share  infonnation  using  data 
primitives  called  information  objects.  Additional  infonnation  on  the  JBI  can  be  found  in 
the  Reference  Implementation  Quick  Start  Guide  (1). 


2.1.1  Information  Obj  ects 

An  information  object  consists  of  two  components,  the  metadata  and  the  payload. 
Information  objects  are  analogous  to  well-defined  emails.  Emails  can  also  be  separated 
into  two  main  parts,  a  message  text  body  and  data  attachments.  The  main  difference 
between  information  objects  and  emails  is  that  information  objects  have  an  exact  format 
that  must  be  followed. 

Metadata  is  written  as  an  Extensible  Markup  Language  (XML)  document  that 
describes  the  content,  quality,  condition,  and  other  characteristics  of  data,  which  is  in  the 
payload,  represented  in  an  information  object,  and  are  often  described  as  “data  about 
data."  Metadata  may  or  may  not  have  a  payload  associated  with  it.  In  Figure  1  is 
presented  a  metadata  weather  related  information  object. 
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<metadata> 

<weather> 

<temperature> 

<value>50</value> 

<scale>F</scale> 

</temperature> 

<location> 

<city>Kansas  City</city> 
<state>Kansas</state> 
</location> 

</weather> 

</metadata> 


Figure  1.  Metadata  Only  Information  Object 

When  the  information  object  has  a  payload  associated  with  it,  then  the  data  in  the 
payload  must  have  an  exactly  defined  format.  This  metadata  then  describes  the  payload 
(i.e.  file  type  and  encoding)  and  any  other  related  information.  Typical  formats  include 
the  exact  type  (e.g.  image,  movies),  and  the  format  of  that  type  (e.g.  Microsoft  Windows 
bitmap  (BMP),  Joint  Photographic  Experts  Group  (JPEG),  and  Audio  Video  Interleave 
(AVI)).  In  Figure  2  is  presented  a  weather  related  information  object  that  includes  a 
payload.  Only  the  metadata  would  be  processed  by  a  JBI  server. 


<metadata> 

<weather> 

<temperature> 

<value>50</value> 

<scale>F</scale> 

</temperature> 

<location> 

<city>Kansas  City</city> 
<state>Kansas</state> 
</location> 

</weather> 

<filetype>jpg</filetype> 

</metadata> 


Payload 


Figure  2.  Metadata  and  Payload  Information  Object 
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Actual  JBI  information  objects  are  more  tightly  defined  than  the  examples  above, 
and  the  exact  specifications  are  presented  in  “JBI  Detailed  Design  Description.”  All 
information  objects  are  described  by  XML  Schemas,  which  define  the  exact  format  of  the 
information  object  metadata.  Before  the  server  accepts  a  publication,  the  server  first 
validates  the  requested  infonnation  object  type  with  the  Schema  stored  in  the  Meta  Data 
Repository  (MDR).  If  a  newly  published  information  object  satisfies  the  stored  Schema 
requirements,  then  that  information  object  is  accepted  and  processed.  If  the  Schema 
requirements  are  not  met,  then  the  information  object  is  simply  dropped. 


2.1.2  Interactions  of  Information  Objects  with  JBI  Servers 

There  are  two  ways  that  information  objects  interact  with  JBI  servers.  The  initial 
interaction  occurs  when  an  information  object  first  arrives  at  the  publication  service.  The 
publication  service  retrieves  the  stored  Schema  definition  and  validates  the  published 
information  object  with  that  Schema  definition.  Validation  is  the  process  of  ensuring  that 
the  right  XML  elements  are  located  at  the  correct  relative  locations,  and  that  the  value 
types  match  the  ones  that  were  mandated  in  the  Schema  definition.  This  evaluation 
process  is  completed  on  a  structural  level,  and  results  in  a  simple  true  or  false  statement 
that  states  whether  or  not  the  infonnation  object  satisfied  the  stored  definition. 

The  second  type  of  information  object/JBI  server  interactions  occurs  during 
Extensible  Metadata  Platform  (XMP)  Path  Language  (XPATH)  predicate  evaluations. 
Subscriber  and  Query  users  supply  XPATH  expressions  which  specify  a  set  of 
requirements  that  define  the  type  of  information  objects  that  are  of  interest.  An  XPATH 
expression  consists  of  one  or  more  predicates,  and  a  predicate  consists  of  clauses.  A 
clause  is  a  logical  value  comparison  that  results  in  a  true  or  a  false.  For  example, 
/metadata/weather/temperature/val ue  <  70  is  a  clause.  A  predicate  is  a  logical 
expression  operating  on  multiple  clauses,  i.e.  /metadata/weather/temperature/value  <  70 
AND  /metadata/weather/temperature/scale  =  “F”. 

In  the  above  examples,  the  exact  element  specified  by  the  XPATH  expression  is 
extracted  from  the  XML  file,  cast  into  the  type  defined  in  the  Schema  definition,  and  then 
compared  to  the  provided  static  value.  Though  the  information  object  values  are  actually 
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extracted  and  interpreted  by  the  JBI  server,  the  values  are  only  used  for  simple 
comparisons,  and  are  never  processed. 

2.1.3  Malicious  Code  Embedded  in  Information  Objects 

The  contents  of  the  information  object  are  never  processed,  and  malicious  code 
that  might  be  embedded  into  an  information  object  can  never  affect  the  JBI  server  itself. 
The  only  foreseeable  avenue  of  adverse  attack  through  an  infonnation  objects  is  by 
exploiting  buffer  overflow  vulnerabilities  in  the  XML  parser  that  extract  element  values 
and  casts  those  elements  into  the  appropriate  types.  These  vulnerabilities  are  not  a 
problem  as  long  as  the  latest  software  updates  for  the  XML  parser  libraries  are  applied. 
Another  means  of  preventing  such  attacks  is  by  ensuring  that  all  variable  length  values 
are  limited  to  a  maximum  length  that  is  less  than  the  maximum  assumed  buffer  length 
used  in  the  XML  parser  libraries. 
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2.2  JBI  Services 


There  are  six  types  of  JBI  services.  These  JBI  services  are  web  administration, 
authentication,  metadata  repository,  publish,  subscribe,  and  query.  Each  of  these  services 
is  described  below.  In  practice  the  JBI  should  remain  operational  at  all  times,  which  will 
ensure  that  users  are  able  to  access  information  and  information  transportation  services 
when  needed  without  time  restrictions. 


2.2.1  Web  Administration  Services 

The  JBI  Web  Administration  Service  is  primarily  used  by  JBI  system 
administrators  to  maintain  user  accounts  and  JBI  servers.  In  addition,  administrators  use 
this  service  to  make  changes  to  the  Information  Object  Repository  (IOR)  and  the 
Metadata  Repository  (MDR).  Changes  made  to  the  MDR,  which  are  immediately 
implemented,  only  affect  the  JBI  server  and  not  the  underlying  system.  The  allowable 
changes  are  displayed  in  the  fonn  of  fill-in  text  boxes  and  expanding  trees  for  privilege 
assignment.  All  of  the  available  commands  are  predefined,  and  are  presented  in  Table  I. 

This  service  allows  administrators  to  log  onto  the  server  to  add  and  remove  user 
and  information  object  descriptions.  Administrators  are  also  able  to  view  the  current 
server  status  and  statistics  and  to  make  changes  to  the  server,  such  as  forcing  a  server  to 
restart  or  flushing  the  repository  contents.  The  following  is  a  list  of  commands  available 
to  the  administrator.  The  get*  commands  require  only  read  access  while  the  others 
require  write  access.  The  resulting  changes  affect  the  JBI  server,  but  not  the  node  on 
which  the  server  is  deployed. 


7 


Table  I.  JBI  Web  Administration  Service  Commands 


command 

purpose 

getAccountsQ 

obtain  a  list  of  user  accounts 

createAccountQ 

create  a  new  user  account 

modifyAccountQ 

make  changes  to  current  account 

deleteAccountQ 

delete  a  current  user  account 

getlnfoObjectTypesQ 

obtain  a  list  of  currently  supported  10  types 

createlnfoObjectTypeQ 

add  a  new  10  type  into  the  JBI  server 

modifylnfoObjectTypeQ 

make  changes  to  an  10  type  (i.e.  update  schema) 

deletelnfoObjectTypeQ 

delete  an  10  type 

getAccountRolesQ 

obtain  Roles  that  an  account  hold 

setAccountRolesQ 

give  an  account  new  Roles 

getlnfoObjectCatalog() 

obtain  a  summary  of  10s  currently  in  persistent  store 

archivelnfoObjectsQ 

place  10s  in  storage,  no  longer  available  for  query 

restorelnfoObjectsQ 

retrive  10s  from  storage 

deletelnfoObjectsQ 

deletes  10s  in  persistent  storage 

Administrators  can  grant  clients  access  to  the  JBI  service  from  a  list  of  individual 
hosts,  from  a  community  of  interest  (COI),  and/or  from  other  appropriate  groupings. 
Only  administrators  are  allowed  to  make  changes  to  the  server.  Administrators  can  be 
further  separated  into  individual  administrative  roles.  All  users  that  have  been  registered 
with  the  JBI  Web  Administrators  are  allowed  service  access,  and  normal  users  are  only 
able  to  view  data  that  they  have  been  granted  access  to.  The  JBI  itself  is  COI  information 
management  system,  and  so  all  users  would  be  part  of  a  common  COI. 

The  administrative  service  uses  the  standard  client  and  server  web  architecture.  A 
client  is  connected  through  a  network  to  the  server  which  resides  on  a  high  perfonnance 
computer  (HPC).  Data  is  transferred  in  a  predefined  text  format.  The  data  is  sent  to  the 
Web  Administration  service  through  a  HyprtText  Transfer  Protocol  Secure  (https)  POST 
request  method.  Data  is  transferred  in  a  predefined  XML  format,  and  is  normally  sent  in 
plain-text.  Secure  Shell  (SSH)  Tunneling  provides  end-to-end  encryption  to  ensure  that 
the  information  remains  confidential.  This  service  listens  on  port  11010  and  maintains 
access  and  connection  control  for  the  JBI  server.  Before  any  user  can  use  a  JBI  service, 
the  user  must  first  authenticate  with  and  obtain  permission  from  this  service.  The 
authentication  service  completes  both  user  authentication  and  authorization  functions. 
User  authentication  ensures  that  the  client  is  a  valid  user  on  the  JBI  system. 
Authorization  ensures  that  the  user  has  the  appropriate  access  rights. 


2.2.2  Authentication  Service 


The  JBI  uses  a  role  based  access  control  system,  which  assigns  each  user  one  or 
more  roles  that  reflect  the  information  object  types  and  services  that  the  user  will  need  to 
access.  Before  a  user  can  use  the  Publish,  Subscribe,  Query  and/or  MDR  services,  the 
user  must  first  authenticate  with  the  Authentication  Service  (Table  II).  After  successful 
authentication,  the  user  is  able  to  connect  to  the  services  requested. 

The  JBI  utilizes  the  Role  Based  Access  Control  (RBAC)  for  fine-grained  security 
policy  enforcement.  Each  infonnation  type  has  a  list  of  roles  that  can  be  set  for  the  user. 
These  roles  fall  into  two  categories,  the  administrator  and  the  user.  User  roles  are  limited 
in  access,  such  that  only  the  JBI  can  be  accessed,  but  the  user  can  not  make  any  changes. 
Administrators,  on  the  other  hand,  can  make  certain  changes. 

Each  infonnation  object  has  the  following  roles  associated  with  it.  A 
username/password  pair  is  sent  to  the  server  for  authentication.  The  information  is  sent 
through  an  http  POST  over  a  Secure  Socket  Layer  (SSL)  connection.  POST  is  an  http 
request  method  which  submits  user  data  to  the  identified  resource.  The  data  is  included 
in  the  body  of  the  request. 

Table  II.  JBI  Authentication  Service  Commands 


Role 

Privilege 

Purpose 

publish 

user 

gives  user  permission  to  publish  this  10  type 

subscribe 

user 

gives  user  permission  to  subscribe  to  this  10  type 

query 

user 

gives  user  permission  to  query  this  10  type 

MDR 

user 

gives  user  read  access  to  the  MDR  service 

MDR 

admin 

gives  user  read  and  write  access  to  the  MDR  service 

admin 

admin 

gives  user  the  privilege  to  make  any  changes  to  this  10  type 

Each  user  can  assume  multiple  roles  at  the  same  time.  The  administrator  can  mix 
and  match  different  roles  to  follow  the  principle  of  least  privilege,  which  allows  the 
minimum  possible  privileges  be  granted  to  permit  a  legitimate  action.  Because  most 
users  will  require  publish  and  subscribe  services,  those  users  will  be  assigned  publish  and 
subscribe  roles  associated  with  the  specific  information  objects  that  are  needed. 
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2.2.3  Metadata  Repository  Service 

An  additional  method  for  making  changes  is  through  the  Metadata  Repository 
(MDR)  Service.  Like  the  Web  Administration  Service,  MDR  service  users  are  able  to 
obtain  a  list  of  the  currently  supported  Information  Objects  and  the  corresponding 
Schema  definitions.  Users  are  also  able  to  update  current  Schema  definitions,  and  to  also 
add  new  ones.  These  changes  are  forwarded  to  the  Infonnation  Management  Staff  (IMS) 
for  review.  Once  the  changes  are  approved,  the  changes  are  finalized  and  the  MDR  is 
updated. 


2.2.4  Publish  and  Subscribe  Services 

The  Publish  and  Subscribe  services  are  the  most  widely  used  basic  services.  The 
Publish  service  is  used  to  send  Infonnation  Objects  to  the  JBI  server  for  distribution.  The 
Subscribe  service  is  used  by  the  user  to  register  interest  in  a  specific  type  of  as 
input/output  (10). 

By  registering  with  the  Subscribe  service,  subscribers  automatically  receive  new 
publications  that  match  the  criteria  provided  to  the  JBI  server.  Criteria  are  presented  to 
the  JBI  server  through  predicates  in  the  form  of  XPATH  expressions.  Information 
objects  are  archived  in  an  Infonnation  Object  Repository  (IOR),  which  is  a  persistent 
database.  The  JBI  server  itself  does  not  decide  whether  or  not  an  10  should  be  archived. 
The  publisher  instructs  the  server  on  what  it  should  do  during  the  initial  Publish 
connection  sequence. 


2.2.5  Query  Services 

Query  is  the  third  basic  JBI  service,  and  is  similar  to  Subscribe,  except  that  the 
archived  Information  objects  are  now  of  interest  to  the  user.  When  new  Information 
objects  are  published  into  the  JBI  server,  the  Query  users  do  not  receive  a  copy  of  the 
Information  objects.  However,  when  the  user  queries  archived  information  and  a  query 
matches  stored  information,  then  the  payload  and  the  meta  data  for  that  10  are  sent  to  the 
requestor. 
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3.  100X  JBI  Security 


The  JBI  Reference  Implementation  was  parallelized  in  this  effort  to  run  more 
rapidly  on  high  performance  computers  and  Linux  clusters.  There  were  additional 
security  concerns  that  were  addressed  so  that  not  only  could  the  100X  JBI  be 
implemented  on  networks,  but  also  so  that  the  new  implementation  could  be  accessed 
interactively. 


3.1  Secure  Shell  Tunneling 

The  six  services  that  were  introduced  in  Section  2  are  the  services  that  users  can 
interact  with.  The  100X  JBI  server  was  deployed  on  High  Performance  Computing 
(HPC)  nodes,  and  the  HPC’s  are  behind  strict  firewalls.  Secure  Shell  (SSH)  Tunneling  is 
used  to  move  Information  objects  through  the  firewalls.  SSH  Tunneling  provides  a 
secure  means  for  clients  to  connect  to  JBI  services  without  having  to  open  up  new  ports 
on  the  firewalls. 

SSH  Tunneling  (2)  is  a  technology  that  allows  packets  from  a  client  port  to  be 
sent  through  an  SSH  connection  to  a  specified  server  port.  One  can  think  of  SSH 
tunneling  as  a  specialized  Virtual  Private  Network  (VPN)  connection.  SSH  Tunneling 
not  only  provides  a  means  for  port  communications  between  a  client  and  a  server,  but  it 
also  provides  security  features. 

Access  to  the  JBI  server  is  restricted  to  users  who  have  SSH  access  to  the 
machine  where  the  server  is  running.  This  means  that  for  the  work  presented  here  access 
to  the  JBI  server  is  limited  to  users  who  have  access  to  the  HPC  center  where  the  JBI 
server  is  hosted.  This  feature  provides  the  first  level  of  defense,  such  that  even  though  a 
JBI  user  account  might  be  compromised,  the  malicious  user  will  not  be  able  to  connect  to 
the  JBI  server  without  that  user  first  having  SSH  access  to  the  HPC  center. 

All  of  the  data  transported  through  the  SSH  tunnel  are  encrypted.  This  is  an 
important  attribute  since  it  provides  confidentiality.  In  the  current  JBI,  users  authenticate 
themselves  by  sending  an  XML  document  with  their  username/password  in  plain  text. 
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The  confidentiality  that  SSH  provides  ensures  that  malicious  users  on  the  network  are  not 
able  to  simply  sniff  the  user’s  account  infonnation. 

All  client  connections  to  the  JBI  rely  on  the  underlying  SSH  connection.  If  the 
SSH  connection  is  broken  or  has  been  purposely  taken  down,  client  connections  will  also 
be  forcefully  terminated. 


Figure  3.  SSH  Tunneling 


In  Figure  3  is  presented  a  more  complete  overview  of  SSH  tunneling.  This  service 
uses  the  HTTPS  internet  protocol,  which  is  HTTP  over  a  Secure  Sockets  Layer  (SSL) 
link,  and  is  configured  to  listen  to  client  connections  on  port  8443.  The  SSH  protocol  is 
also  used  to  provide  a  tunnel  between  the  client  and  the  JBI  server. 

3.2  Information  Objects 

The  100X  JBI  operates  on  generic  Information  Objects,  and  only  non-sensitive 
information  was  managed  in  this  prototype  100X  JBI  development.  Proper  security 
precautions  must  be  taken  before  any  sensitive  material  is  to  be  managed  by  the  100X 
JBI.  The  level  at  which  the  service  is  run  (user  privileges),  the  level  of  access  required 
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(access  to  root  or  any  other  non-user  accounts,  indirect  access  via  setuid,  setgid,  or  other 
means),  and  the  security  level  of  the  network  over  which  the  software  is  installed  must  be 
considered. 


In  Figure  4  is  presented  a  diagram  of  data  flow  for  the  100X  JBI.  This  is  a 
standard  web  service  that  uses  the  request  and  reply  interaction  scheme.  The  only 
difference  between  this  service  and  a  normal  HTTP  web  administration  service  is  that 
nonnal  HTTPS  servers  listen  on  port  443  and  this  service  listens  on  port  8443. 
Furthermore,  access  to  this  service  must  be  preceded  by  obtaining  access  to  the  node  on 
which  the  service  is  running.  In  general  this  means  that  the  user  must  obtain  a  Kerberos 
ticket  and  then  establish  a  SSH  connection  and  setup  port  forwarding. 


3.3  Authentication 


The  Authentication  service  is  central  to  the  JBI,  and  is  used  by  all  the  other  JBI 
services  to  ensure  that  users  are  pennitted  to  exercise  whatever  task  they  are  requesting. 
Depending  on  the  requested  service,  authentication  takes  on  a  different  shape.  Publish, 
Subscribe,  Query  and  MDR  users  must  all  first  authenticate  directly  with  the 
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Authentication  to  obtain  a  Connection  ID.  Once  a  Connection  ID  is  obtained,  the  user 
can  then  connect  with  the  specific  service  of  interest. 

Web  Administration  users  authenticate  using  a  less  direct  method  by  sending  their 
credentials  to  the  Web  Administration  service.  This  service  then  uses  that  infonnation  to 
authenticate  the  administrator  with  the  Authentication  service. 

3.3.1  Web  Administration 

A  Web  Administration  user  sends  authentication  information  to  the  Web 
Administration  service,  and  that  service  in  turn  authenticates  the  user  with  this 
Authentication  service.  Authorization  is  done  on  a  per-command  basis.  Whenever  the 
user  initiates  one  of  the  commands  described  in  Section  2.2.1,  this  Authentication  service 
is  consulted  to  ensure  that  the  user  has  the  proper  permissions  to  complete  the  command. 

3.3.2  Users 

A  user  in  this  category  authenticates  with  this  service  directly.  Upon  successful 
authentication,  the  user  will  then  establish  a  connection  sequence  with  the  specific 
service  he  wants  to  use.  The  service  will  take  the  user’s  infonnation  and  consult  with  this 
service  to  ensure  that  the  he  is  authorized  to  use  the  requested  service. 


3.3.3  The  Publish,  Subscribe,  Query  and  MDR 

The  publish,  subscribe,  query  and  MDR  services  follow  a  similar  procedure  to 
that  of  the  user  for  establishing  connections  with  users. 


3.3.4  Kerberos 

Firewalls,  while  offering  security  from  deleterious  and/or  malicious  effects  from 
outside  a  network,  restrict  the  flexibility  of  an  internet  based  network.  To  use  the  100X 
JBI  as  an  information  management  system  in  the  field,  the  distributed  users  would 
necessarily  be  outside  the  firewall.  Because  the  100X  JBI  was  developed  to  run  on  the 
Distributed  Interactive  High  Perfonnance  Computing  Testbed,  which  is  based  upon  the 
Defense  Research  and  Engineering  Network  (DREN),  there  are  firewalls  rules  imposed 
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by  the  High  Performance  Computing  Modernization  Program  (HPCMP)  that  must  be 
followed.  To  use  any  of  the  100X  JBI  services,  the  user  must  first  be  authenticated  to 
access  the  assets  of  the  DIHT. 

Kerberos  is  a  network  authentication  protocol,  and  was  designed  to  provide  strong 
authentication  for  client/server  applications  by  using  secret-key  cryptography.  Users 
must  be  running  Kerberos  software  on  the  local  computer  and  have  a  one-time  password 
SecurlD  card  issued  by  the  DoD’s  High  Performance  Computing  Modernization  Program 
(https://kirby.hpcmp.hpc.mil/).  The  SecurelD  utilizes  a  separate  password  or  Personal 
Identification  Number  (PIN)  and  an  authenticator 

(http://www.rsasecurity.com/node.asp/idM  156).  The  SecurelD  card  supplements  the 
password  based  authentication  mechanism  by  adding  the  additional  requirement  of  a 
passcode  derived  from  a  time-dependant  code  generated  by  the  SecurelD  card.  This  tests 
something  you  know  and  something  you  have,  and  are  two  categories  or  classes  of 
authentication  mechanisms. 

Kerberos  is  a  two  stage  process.  First  the  user  authenticates  with  the 
Authentication  Service,  which  is  run  on  a  Kerberos  Server.  The  server  sends  the  user  a 
Ticket  Granting  Ticket  (TGT).  When  the  user  wants  access  to  a  computer  on  the  DIHT, 
the  user  sends  a  TGT  to  a  Ticket  Granting  Service  (TGS),  and  the  TGS  returns  a  Session 
Service  Ticket  (SST).  The  user  then  uses  the  SST  to  authenticate  and  access  the  DIHT 
computer.  The  Authentication  Service  and  the  Ticket  Granting  Service  are  two  different 
entities,  and  can  be  hosted  on  the  same  server. 

If  more  than  one  computer  is  to  be  used  on  the  DIHT,  then  SST’s  for  each 
computer  must  be  obtained.  An  issued  SST  has  a  set  period  of  time  before  it  expires.  In 
cases  where  extended  use  is  required,  either  new  tickets  will  need  to  be  obtained,  or 
arrangements  need  to  be  made  for  longer  duration  tickets. 
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4.0  100XJBI  ARCHITECTURE 


The  architecture  of  the  100X  JBI  project  is  based  on  the  AFRL  JBI  Reference 
Implementation  1.2.  The  JBI  Reference  Implementation  architecture  adopted  software 
methodologies  that  provided  maximum  flexibility  for  experimenting  with  new  features, 
and  was  independent  of  performance.  The  conceptual  framework  for  a  net-centric 
pub/sub  system  included  a  standard  Common  Application  Programming  Interface 
(CAPI).  This  Reference  Implementation  was  based  on  a  proof  of  concept  that  has  been 
implemented  to  interoperate  in  a  net-centric  system  environment. 

The  100X  JBI  goals  included  a  two-order  of  magnitude  speedup  over  the 
reference  implementation.  To  achieve  this  goal,  a  two  fold  approach  was  implemented, 
which  focused  on  converting  JAVA  codes  to  C++,  and  then  parallelizing  the  codes.  The 
architecture  of  the  100X  JBI  is  based  upon  the  JBI  1.2  Reference  Implementation 
architecture,  which  is  presented  in  Figure  5. 


Figure  5.  Overview  of  the  100X  JBI  architecture. 
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Information  objects  arrive  at  the  publisher  interface,  are  sent  to  the  broker  which 
sends  the  publication  to  repository,  and/or  matches  the  publication  against  earlier 
subscriptions,  and  if  a  match  is  found,  forwards  the  information  object  to  the  subscriber 
who  requested  it.  If  a  query  arrives  at  the  publisher  interface,  that  query  is  compared  to 
the  stored  publications  in  the  repository,  and  if  a  match  is  found,  the  resultant  publication 
is  forwarded  to  the  requestor. 


4.1  Communications 

The  first  step  in  the  implementation  of  the  100X  JBI  was  to  speed  up  the  basic 
communications.  The  configuration  of  the  JBI  reference  implementation  was  changed  to 
bypass  the  Java  Messaging  Services  (JMS)  layer,  and  the  Java  JNI  (Java™  Native 
Interface)  communications  software  was  replaced  with  a  TCP/IP  socket  implementation 
in  C++.  The  communications  changes  affected  the  JBI  core  services  implementation  as 
well  as  the  CAPI  library  interface. 

The  JniConnection  and  JniConnectionService  were  the  first  Java  components  of 
the  reference  implementation  that  were  replaced  in  the  100X  JBI.  The  initial  change  in 
communications  software  made  a  significant  improvement  in  speedup  performance.  As 
expected,  the  C++  compiled  code  performed  faster  that  the  Java  Virtual  Machine-based 
system.  These  initial  results  were  encouraging,  and  language  conversion  from  Java  to 
C++  was  one  of  the  principal  approaches  used  for  100X  JBI  performance  enhancements. 

JniConnection  is  the  Java  Native  Interface  (JNI)  wrapper  to  call  the  C++  JBI 
client  libraries  from  Java.  JniConnectionService  is  the  Java  Native  Interface  C++  code 
that  enabled  100X  JBI  clients  in  C++  or  Java  to  communicate  to  the  original  JBI  Java 
server  code  that  provided  the  security  framework. 
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4.2  Publishing 


The  publication  services  were  the  next  piece  of  the  JBI  that  was  considered  for 
high  performance  implementation.  Initially,  publisher  sequence  support  was  added  to  the 
C++  JNI  communications  interface,  and  a  publication  catcher  component  was  added  to 
the  100X  JBI  server.  The  CAPI  was  tested  to  validate  the  connections  to  remote  JBI 
servers.  The  publication  catcher  received  publications  from  CAPI  clients  and  forwarded 
the  publications  to  the  brokers  with  which  it  was  configured  to  interoperate. 

Publication  services  required  Extensible  Markup  Language  (XML)  libraries  to  be 
added  to  the  implementation  along  with  some  other  basic  JBI  utilities  for  constructing 
messages  and  for  control  over  exceptions.  Initial  testing  of  the  Field  Programmable  Gate 
Array  (FPGA)-based  XML  filtering  support  was  conducted  by  filtering  messages  passing 
through  the  pubcatcher,  although  the  broker  had  not  yet  been  implemented. 


4.3  Subscribing 

Subscription  services  were  added  to  the  100X  JBI.  The  changes  involved  the  JNI 
implementation,  the  core  services  and  the  CAPI.  Subscriber  services  included  registering 
subscriptions  with  a  broker  and  establishing  “object  available  callbacks”  to  receive 
notification  when  brokers  discovered  publications  that  match  the  subscriptions. 


4.4  Brokering/Disseminating 

Brokering  services  match  publications  with  interested  subscribers,  and  metadata 
comparisons  determine  the  desirability  of  publications  for  subscribers.  Registered 
subscriptions  that  match  a  received  publication  are  forwarded  to  the  user  through  the 
object  available  callback  subscriber  method.  The  broker  uses  a  disseminator  process  to 
distribute  publications  to  users,  which  also  avoids  delays  in  sending  requested 
publications  over  possibly  slow  network  connections.  The  broker  used  shared  memory  to 
enhance  it’s  perfonnance. 


18 


4.5  JBI  Connectors 


The  parallel  distributed  high  performance  100X  JBI  used  “JBI  Connectors”  to 
handle  distributed  operations.  In  a  100X  JBI  information  management  system,  there  may 
be  several  JBI  servers,  each  implementing  publication  and  subscription  services.  100X 
JBI  connectors  forward  messages  to  remote  JBI’s,  where  a  subscription  may  be  brokered. 
The  Message  Passing  Interface  (MPI)  was  used  to  improve  the  performance  and  allow 
100X  JBI  server  scaling  for  high  performance  computer  implementations.  Peer  services 
allowed  publication  catchers  to  forward  publications  to  remote  100X  JBI  servers.  This 
architecture  allowed  multiple  publication  catchers  for  each  JBI. 


4.6  Query  and  Archive 

Publications  can  be  available  for  users  who  are  not  connected  when  a  publication 
of  interest  is  published.  Users  can  request  archival  services  when  publishing.  A  query 
function  allows  users  to  request  archived  documents  using  a  syntax  that  was  similar  to  the 
subscription  syntax.  Along  with  the  archival  and  query  services,  the  ability  to  handle 
payloads  and,  for  enhanced  performance,  large  memory-based  payloads  with  metadata 
was  added  to  the  100X  JBI. 


4.7  100X  JBI  Architecture 

In  Figure  6  is  presented  a  more  detailed  overview  of  the  100X  JBI  architecture. 
At  the  top  of  the  figure  is  shown  that  the  authorization  credentials  for  publish  requests 
and  subscription  requests  are  sent  to  the  Connector  Manager,  which  then  connects  to  the 
Information  Peer  List.  A  more  complete  description  of  this  process  is  presented  in 
Section  5.2. 

When  incoming  publications  are  received  from  other  HPC’s  and/or  from 
authorized  clients,  these  publications  are  temporarily  held  in  the  Publisher  Catcher. 
There  can  be  from  one  to  m  Publisher  Catchers.  When  there  is  more  than  one  Publisher 
Catcher,  then  each  Publisher  Catcher  is  located  on  a  separate  processor.  The  publications 
in  the  Publisher  Catchers  are  sent  to  a  Connector  (a  single  processor),  which  then 
transmits  a  replicate  of  the  publication  to  from  zero  to  three  other  HPC’s.  If  other  HPC’s 
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are  being  transmitted  to,  then  the  system  has  redundancy.  The  publication  is  also  sent  to 
the  next  available  Broker. 

There  can  be  from  one  to  n  numbers  of  Brokers,  and  if  more  than  one  Broker  is 
being  used,  then  each  Broker  is  on  a  separate  processor.  A  Broker  compares  the  elements 
of  the  meta  data  with  the  predicates  for  each  subscription  that  has  been  previously 
submitted.  When  a  match  is  found,  then  the  corresponding  publication  is  sent  by  the  next 
available  Disseminator  (one  to  k  total  Disseminators),  which  then  transmits  the 
publication  to  the  client  that  had  submitted  the  subscription. 
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Figure  6.  Detailed  Overview  of  the  100X  JBI  architecture 
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5.  100X  JBI  System  Sequence  Diagrams 

In  this  chapter  Unified  Modeling  Language  (UML)  System  Sequence  Diagrams 
(SSD’s)  are  presented  for  the  100X  JBI  architecture.  For  each  use  case  scenario,  the 
events  that  the  user  generates,  the  order  that  those  events  are  generated  in,  and  the 
responses  of  the  system  to  the  generated  event  are  listed. 

5.1  Publication 

In  Figure  7  is  presented  the  SSD  diagram  for  the  100X  JBI  publication  service. 
After  the  Kerberos  ticket  is  issued  to  the  publisher,  an  SSH  connection  is  requested 
through  Port  22  of  the  high  performance  computer  or  Linux  cluster  on  which  the  JBI 
server  is  located.  When  the  connection  is  granted,  a  SSH  tunnel  is  created  from  port 
11010  on  the  local  host  to  Port  11010  on  the  HPC.  Authentication  with  the  JBI  server  is 
achieved  through  this  tunnel.  A  unique  Connection  ID  Number  is  issued  by  the  JBI 
server  and  transmitted  back  to  the  publisher  through  SSH  tunneling. 

The  setup  of  the  publication  sequence  then  occurs,  in  which  the  Connection  ID 
Number,  notification  that  this  is  a  publication,  the  10  type,  and  the  IOVer  are  SSH 
tunneled  to  Port  11011  of  the  JBI  server.  The  JBI  server  then  issues  a  unique  Publication 
Sequence  ID  Number  to  the  publisher  by  SSH  tunneling. 

The  publisher  then  uses  this  unique  Publication  Sequence  ID  Number  in  future 
communications  with  the  server.  If  the  publisher  chooses  to  publish  an  information 
object,  the  whole  object  is  serialized  and  sent  through  the  SSH  tunnel  to  the  server  along 
with  the  Sequence  ID  Number.  Once  the  server  acknowledges  that  the  publication  has 
met  the  schema  standards,  a  unique  Object  ID  is  sent  back  to  the  publisher.  In  the  case 
when  acknowledgements  are  not  request  by  the  publisher,  a  unique  ID  is  not  sent  back  to 
the  publisher. 
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Figure  7.  Publish  System  Sequence  Diagram 
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5.2  Subscribe 


In  Figure  8  is  presented  the  SSD  for  the  100X  JBI  subscribe  service.  After  the 
Kerberos  ticket  is  issued  to  the  subscriber,  an  SSH  connection  is  requested  through  Port 
22  of  the  high  performance  computer  or  Linux  cluster  on  which  the  JBI  server  is  located. 
When  the  connection  is  granted,  an  SSH  tunnel  is  created  from  port  11010  on  the  local 
host  to  Port  11010  on  the  HPC.  Authentication  with  the  JBI  server  is  achieved  through 
this  tunnel.  A  unique  Connection  ID  Number  is  issued  by  the  JBI  server  and  transmitted 
back  to  the  subscriber  through  SSH  tunneling. 

The  setup  of  the  subscription  sequence  then  occurs,  in  which  the  Connection  ID 
Number,  notification  that  this  is  a  subscription,  the  10  type,  the  IOVer,  and  the  XPATH 
predicate  are  SSH  tunneled  to  Port  1 1012  of  the  JBI  server.  The  JBI  server  then  registers 
the  predicate  for  future  matching  of  incoming  publications.  The  predicate  is  stored  in  a 
lookup  table  that  any  broker  can  access. 

Should  a  publication  arrive  at  the  JBI  server  which  matches  the  predicate  data  of 
the  subscription,  then  the  sequence  ID  of  the  matching  publication  is  delivered  to  the 
subscriber  by  SSH  tunneling  by  the  JBI  server.  The  subscriber  immediately  receives  the 
IO  if  the  10  is  less  than  128x8  in  size.  If  the  size  of  the  10  is  large  than  this,  then  the 
subscriber  must  send  an  acknowledgement  back  to  the  server  before  the  server  sends  the 
IO  to  the  subscriber.  Once  the  subscription  is  completed,  the  subscriber  requests  to  the 
JBI  server  that  the  subscription  sequence  be  destroyed  through  Port  11012.  The  JBI 
server  acknowledges  the  request  to  the  subscriber,  and  the  subscriber  SSH  tunnels  to  the 
JBI  server  that  Ports  11010  and  11012  are  to  be  closed.  Through  SSH  tunneling  the  JBI 
sever  acknowledges  to  the  subscriber  that  the  ports  are  closed,  and  the  subscription 
sequence  is  completed. 
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Figure  8.  Subscribe  System  Sequence  Diagram 
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5.3  Query 


In  Figure  9  is  presented  the  SSD  for  the  100X  JBI  query  service.  The  query  user 
must  first  authenticate  to  request  a  Kerberos  ticket  from  the  Kerberos  Server.  When  the 
Kerberos  ticket  is  issued  to  the  query  user,  an  SSH  connection  is  requested  through  Port 
22  of  the  high  performance  computer  or  Linux  cluster  on  which  the  JBI  server  is  located. 
When  the  connection  is  granted,  then  SSH  tunneling  authentication  is  achieved  from  the 
JBI  server  through  Port  11010.  A  unique  Connection  ID  number  is  issued  by  the  JBI 
server  to  the  subscriber  through  SSH  tunneling. 

Next  the  setup  of  the  query  sequence  occurs,  in  which  the  connection  ID  number, 
notification  that  this  is  a  query,  the  10  type,  the  IOVer,  and  the  XPATH  are  SSH  tunneled 
to  Port  11013  of  the  JBI  server.  The  JBI  server  then  issues  a  Unique  Sequence  ID 
Number  for  the  query  by  SSH  tunneling  to  the  query  user.  The  JBI  server  then  checks 
the  JBI  repository  for  matching  predicates. 

A  list  of  matching  information  objects  is  sent  back  to  the  query  user  and  the  query 
user  determines  which  resulting  information  objects  he  wants. 
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Figure  9.  Query  System  Sequence  Diagram 
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5.4  Meta  Data  Repository 


In  Figure  10  is  presented  the  SSD  for  the  100X  JBI  Meta  Data  Repository  (MDR) 
service.  The  user  of  the  MDR  must  first  authenticate  to  request  a  Kerberos  ticket  from 
the  Kerberos  Server.  When  the  Kerberos  ticket  is  issued  to  the  user,  an  SSH  connection 
is  requested  through  Port  22  of  the  high  performance  computer  or  Linux  cluster  on  which 
the  JBI  server  is  located.  When  the  connection  is  granted,  then  SSH  tunneling 
authentication  is  achieved  from  the  JBI  server  through  Port  11010.  A  unique  Connection 
ID  number  is  issued  by  the  JBI  server  to  the  user  through  SSH  tunneling. 

The  JBI  server  then  searches  for  a  matching  Schema.  If  a  matching  Schema  is 
found,  then  the  user  is  notified  that  a  matching  schema  was  located  by  SSH  tunneling. 
The  user  then  makes  a  Schema  request  over  Port  11014  by  SSH  tunneling,  and  the  JBI 
server  SSH  tunnels  back  the  SCHEMA.  The  user  then  sends  to  the  JBI  server  over  Port 
11014  by  SSH  tunneling  a  request  to  destroy  the  MDR  sequence  request,  and  the  JBI 
server  by  SSH  tunneling  replies  that  the  MDR  sequence  request  has  been  destroyed. 
Next  the  setup  of  the  MDR  sequence  occurs,  in  which  the  connection  ID  number, 
notification  that  this  is  an  MDR,  the  IO  type,  the  IOVer,  and  the  Update  are  SSH 
tunneled  to  Port  11014  of  the  JBI  server.  The  JBI  server  then  sends  a  unique  MDR 
Sequence  ID  number  by  SSH  tunneling  to  the  user.  The  user  then  sends  to  the  JBI  server 
the  new  Schema  by  Port  1 1014  by  SSH  tunneling,  and  the  JBI  server  acknowledges  back 
to  the  user  by  SSH  tunneling  the  receipt  of  the  Schema  request.  The  administrator  of  the 
JBI  server  is  notified  that  a  new  schema  has  been  submitted  for  review.  The  user  then 
SSH  tunnels  the  JBI  server  to  destroy  the  MDR  Sequence  ID  over  Port  11014,  and  the 
JBI  server  acknowledges  to  the  user  by  SSH  tunneling  that  it  was  destroyed.  The  user 
then  requests  by  SSH  tunneling  that  the  connection  be  closed  through  Port  11010,  and  the 
JBI  server  acknowledges  by  SSH  tunneling  that  the  connection  has  been  closed. 

Should  a  publication  arrive  at  the  JBI  server  which  matches  the  predicate  data  of 
the  subscription,  then  the  sequence  ID  of  the  matching  publication  is  delivered  to  the 
subscriber  by  SSH  tunneling  by  the  JBI  server.  The  subscriber  then  requests  to  the  JBI 
server  that  the  subscription  sequence  be  destroyed  through  Port  11012.  The  JBI  server 
acknowledges  the  request  to  the  subscriber,  and  the  subscriber  SSH  tunnels  to  the  JBI 
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server  that  Port  11010  is  to  be  closed.  Through  SSH  tunneling  the  JBI  sever 
acknowledges  to  the  subscriber  that  the  port  is  closed,  and  the  subscription  sequence  is 
completed. 
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Figure  10.  Meta  Data  Repository  System  Sequence  Diagram 


29 


6.  Distributed  Interactive  HPC  Testbed 


6.1  Overview 

The  Distributed  Interactive  HPC  Testbed  (DIHT)  is  an  experimental  testbed  that 
is  hosted  on  the  Defense  Research  and  Engineering  Network  (DREN) 
(http://www.hpcmp.hpc.mil/Htdocs/DREN/index.html  ),  and  was  recently  implemented 
by  the  Information  Directorate  of  the  Air  Force  Research  Laboratory  to  provide  scientists 
and  engineers  with  capabilities  for  high  performance  computing  that  are  distributed  over 
a  wide  geographic  area  with  real-time  interactive  responsiveness.  The  mission  of  the 
DoD’s  High  Performance  Computing  Modernization  Program  (HPCMP),  which  sponsors 
the  DREN,  is  to  develop  high  performance  computing  (HPC)  capabilities  within  the 
DoD’s  Research,  Development,  Test  &  Evaluation  (RDT&E)  community. 

The  HPCMP  accomplishes  its  mission  by  providing  access  to  HPC  machines  to 
the  DoD  community  through  three  categories  of  sites.  The  Major  Shared  Resource 
Centers  (MSRC’s)  house  large  supercomputers  and  provide  computing  cycles  to  users 
across  the  nation.  The  Distributed  Centers  (DC’s)  deploy  more  modest  systems,  satisfy 
more  local  needs,  and  enable  the  host  organizations  to  stay  at  the  forefront  of  HPC 
technology.  The  DC’s  strive  to  develop  new  software  applications  and/or  evaluate 
advanced  computing  and  communications  technologies.  There  are  also  DoD  User  Sites 
that  are  geographically  dispersed  on  the  DREN.  The  Distributed  Interactive  HPC  Test 
Bed  (DIHT)  is  a  collaboration  of  one  MSRC,  one  DC  and  2  User  Sites  that  provide 
interactive  and  distributed  capabilities.  Additionally,  other  distributed  authorized  users 
of  the  DIHT  need  only  connect  to  the  DREN  to  gain  access  to  the  testbed. 


6.2  Linux  Clusters 

In  Figure  1 1  is  presented  the  locations  of  the  distributed  Linux  clusters  that  are 
integrated  together  on  the  DIHT,  and  a  more  complete  description  of  the  clusters  is 
presented  in  Table  III. 
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Figure  11.  Distributed  Interactive  HPC  Testbed 


Table  III.  DIHT  Linux  Clusters 


Linux  Computer 

Cluster 

Location 

Processors 

Memory 

I/O 

Powell 

ARL  MSRC 
Aberdeen,  MD 

128  node  dual 
3.06MHz  Xeon 

2  GB  DRAM 

64  GB  disk/node 

Myrinet  &  GigEnet 

100MB  Backplane 

Mach2 

ASC  MSRC 
Dayton,  OH 

24  node  dual  2.66 
GHz  Xeon 

4  GB  DRAM 

80  GB  disk/node 

Dual  GigEnet 

Coyote 

AFRL 

Rome,  NY 

26  node  dual  3.06 
GHz  Xeon 

6  GB  DRAM 

400  GB  disk/node 

Dual  GigEnet 

HHPC 

AFRL 

Rome,  NY 

48  node  dual  2.6 

GHz  Xeon  Wildstar 

II  FPGA  cards 

2  GB  DRAM 

64  GB  disk/node 

2Gb  Myrinet  & 
GigEnet 

Seafarer 

SSCSD 

San  Diego,  CA 

24  node  dual  3.06 
GHz  Xeon 

4  GB  DRAM 

80  GB  disk/node 

Dual  GigEnet 

Koa 

MHPCC 

Maui,  HI 

128  node  dual  Xeon 

4  GB  DRAM 

80  GB  disk/node 

Shared  file  system 

Dual  GigEnet 
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6.2.1  Mach  2 


At  the  Aeronautical  Systems  Center  (ASC)  MSCR  is  the  Linux  cluster  Mach2, 
which  is  a  24  node,  2.66  GHz  dual  Intel  Xeon  cluster  with  4  GB  Dynamic  Random 
Access  Memory  (DRAM)  and  80  GB  disk  per  node  and  dual  gigabit  Ethernet  (GigEnet) 
interconnection  fabric.  This  system  uses  Red  Hat  Linux  Enterprise  3  as  its  operating 
system. 


6.2.2  Powell 

At  the  Anny  Research  Laboratory  (ARL)  MSCR  is  located  the  Linux  Cluster 
Powell,  which  is  a  128  node,  3.06  GHz  dual  Intel  Xeon  cluster  with  2  GB  DRAM  and  64 
GB  disk  per  node  with  Myrinet  and  GigEnet  interconnection  fabrics  and  a  100  MB 
backplane.  Powell  is  comprised  of  four  different  types  of  nodes:  compute,  storage,  login, 
and  management.  All  nodes  contain  dual  Intel  XEON  processors  integrated  with  an  Intel 
7501  chipset,  512k  of  Level  2  cache  and  2  GB  of  ECC-protected  memory.  The  compute 
and  login  nodes  have  3.06  GHz  processors  while  all  others  have  2.4  GHz  processors.  The 
128  compute  nodes  total  256  processors  and  256  GB  of  memory  with  a  peak  system 
performance  of  1.566  Teraflops.  All  compute  nodes  have  a  Myrinet  2000  interconnect 
which  offers  2  Gbps  bandwidth  into  and  out  of  each  node.  The  interconnect  is  non- 
blocking  and  supports  MPI  communications.  The  latency  on  the  switch  is  less  than  6ps  or 
250  MB/sec  bi-sectional  bandwidth. 

The  storage  nodes  provide  the  cluster  with  an  aggregate  of  10  TB  disk  space, 
divided  among  5  file  systems  that  are  globally  accessible  and  offer  scalable  parallel 
performance  via  Sistina's  Global  File  System  (GFS).  All  of  the  nodes  share  a  gigabit 
Ethernet  network  which  provides  a  high-performance  communications  conduit  between 
the  storage  nodes  and  the  compute  and  login  nodes.  This  network  also  connects  to  the 
ARL  MSRC  backbone  network,  providing  a  high-performance  path  to  the  mass  storage 
archival  system.  In  addition,  all  nodes  have  access  to  the  Defense  Research  and 
Engineering  Network  (DREN). 
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6.2.3  Koa 


The  128  node,  dual  3.06  GHz  Intel  Xenon  Linux  cluster  called  Koa  is  located  at 
the  Maui  High  Performance  Computing  Center.  This  cluster  has  4  GB  of  memory  per 
node,  and  the  nodes  are  interconnected  via  gigabit  Ethernet. 


6.2.4  Coyote 

At  the  Information  Directorate  at  the  Air  Force  Research  Laboratory  is  the  Coyote 
Linux  Cluster,  a  26  node  dual  3.06  GHz  Intel  Xeon  cluster  with  4  GB  DRAM  and  400 
GB  disk  per  node  and  a  gigabit  Ethernet  (GigEnet)  interconnection  fabric. 


6.2.5  Heterogeneous  HPC 

Also  at  the  Information  Directorate  is  the  Heterogeneous  HPC,  a  48  node  dual  2.6 
GHz  Intel  Wildstar  II  Field  Programmable  Gate  Array  (FPGA)  cluster  with  2  GB  DRAM 
and  64  GB  disk  per  node  and  2  Gb  Myrinet  and  GigEnet  interconnection  fabric. 


6.2.6  Seafarer 

At  the  Space  and  Naval  Warfare  Systems  Center,  San  Diego  (SSCSD)  is  the 
Linux  cluster  Seafarer,  which  is  identical  to  Mach2. 


6.3  Testbed  Applications 

Among  the  early  interactive  applications  of  this  testbed  has  been  the  testing  of  the 
100X  JBI  information  management  system.  Typical  usage  of  the  HPCMP’s  HPC 
resources  is  via  batch  mode  -  where  users  request  some  amount  of  processing  time  on  an 
HPC  resource,  submit  their  jobs  to  the  resource’s  batch  queue,  and  wait  for  their  jobs  to 
reach  the  top  of  the  queue  for  execution.  This  could  take  several  hours  or  even  several 
days  -  depending  upon  many  factors.  There  exists  a  need  within  the  DoD  HPC  user 
community  for  an  interactive  capability  in  which  the  user  requires  the  result  within 
minutes  or  even  seconds. 
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One  prominent  justification  for  pursuing  distributed  computing  (also  known  as 
grid  computing,  networked  computing,  or  meta-computing)  is  to  leverage  computer 
resources  that  are  perhaps  tens  to  hundreds  of  times  more  powerful  than  is  typically 
housed  at  a  single  facility.  But  there  are  also  two  other,  equally  strong  justifications  - 
namely:  (1)  In  typical  battlespace  environments,  the  data  (e.g.,  sensors)  are  inherently 
geographically  dispersed;  hence  distributing  the  computing  resources  close  to  the  data 
saves  the  bandwidth  and  latency  required  to  communicate  them  to  a  centralized 
processor.  And  (2)  the  “players  in  the  game”  (i.e.,  the  warfighters)  are  inherently 
dispersed  hence  having  multi-source  data-fusion  and  battle-planning  processors  close  to 
the  local  decision  makers  makes  very  good  sense. 

It  should  be  noted  that  this  testbed  is  inherently  well  suited  to  explore  paradigms 
for  network-centric  warfare  (whose  requirements  are  inherently  highly  distributed  and 
interactive).  It  is  expected  that  experimental  results  will  benefit  the  Air  Force’s  C2 
Constellation  and  Joint  Battlespace  Infosphere  programs,  the  Navy’s  FORCEnet 
program,  and  the  Army’s  Future  Combat  Systems  program. 
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7.0  Core  Service  Experimentation 

Several  experiments  were  designed  and  run  to  determine  the  capacity  and  speed 
of  the  100X  JBI  architecture,  and  the  results  of  these  determinations  were  compared  to 
the  JBI  Reference  Implementation  1.2.  The  capacities  of  publisher  catchers  and  brokers 
to  process  Information  Objects  were  determined,  and  then  the  capacity  of  the  overall 
100X  JBI  to  process  and  disseminate  information  objects  were  detennined  as  a  function 
of  the  complexities  of  predicate  complexities. 


7.1  Publisher  Catcher  Capacity 

The  publisher  catcher  receives  the  incoming  information  objects  from  clients 
and/or  other  hpc’s,  and  holds  those  publications  in  a  queue  until  the  next  broker  is 
available,  at  which  time  that  publication  is  sent  to  that  available  broker  for  processing 
(Figure  12).  The  publisher  catcher  sends  both  the  meta  data  and  the  payload  to  the  next 
broker  when  the  total  size  of  the  incoming  publication  is  less  than  128  kbytes.  If  the  total 
size  is  greater  that  128  kbytes,  then  the  payload  is  sent  to  memory,  and  only  the  meta  data 
is  sent  to  the  broker.  As  a  broker  processes  an  Infonnation  Object,  whenever  there  is  a 
match,  the  Infonnation  Object  is  sent  to  the  disseminator,  which  forwards  that 
Information  Object  to  the  requestor. 


Next 


^Broker  1 
jBroker  2 


Available  .^Broker  3 
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Figure  12.  100X  JBI  Architecture  Publisher  Catcher  Capacity 
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7.1.1  Experiment 


This  experiment  was  designed  to  detennine  the  number  of  infonnation  objects 
that  can  be  processed  per  time  unit  by  a  single  Publisher  Catcher.  It  was  conducted  on 
the  Heterogeneous  High  Performance  Computer,  and  the  information  object  sizes  were  2 
kbytes  each.  Additional  Publisher  Catcher  experimental  parameters  are  presented  in 
Table  IV,  and  other  more  specific  experimental  details  are  presented  in  Table  V. 

In  this  experiment  incoming  information  objects  were  submitted  to  one  Publisher 
Catcher.  The  infonnation  object  was  then  sent  through  a  queue  to  the  next  available 
Broker.  The  experiment  consisted  of  several  iterations  in  which  the  number  of  brokers 
was  increased  from  1  to  20,  and  the  numbers  of  information  objects  processed  per  second 
were  recorded.  For  this  experiment  each  broker  was  on  a  separate  processor. 


Table  IV.  Publisher  Catcher  Parameters 


Computer 

Heterogeneous  High  Perfonnance 

Computer  (HHPC) 

Processor  speed 

2.6  GHz 

Publication  Size 

2  kbytes 

Publishing  Processors 

1 

Broker  Processors,  n 

1,2,  4,  8,  16,20 

Disseminator  Processors 

1 
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Table  V.  Publisher  Catcher  Experimental  Conditions 


#Run 

#brokers 

#pub  catchers 

#nodes 

1 

1 

1 

1 

2 

2 

1 

2 

3 

4 

1 

3 

4 

8 

1 

5 

5 

16 

1 

9 
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7.1.2  Results 


The  results  of  the  experiment  to  determine  the  Publisher  Catcher  capacity  are 
presented  in  Figure  13.  With  a  one  processor  brokering  system  (Run  1),  8,200 
information  objects  of  2  kbyte  size  were  processed  in  one  second.  Increasing  the  number 
of  brokers  to  two  (Run  2)  increased  the  number  of  information  objects  being  processed  to 
12,200  per  second.  Marginal  increases  to  13,000  information  objects  per  second  were 
realized  by  increasing  the  number  of  brokering  processors  beyond  two  (Runs  3-6).  From 
the  experimental  results  presented  in  Figure  13,  the  Publisher  Catcher  capacity  was 
determined  to  be  13,000  information  objects  per  second,  which  corresponded  with  the 
processing  of  26  Mbytes  per  second. 


Information  Object  Size  =  2  kbytes 


Number  of  Brokers 


Figure  13.  Information  Objects  Processed  v.  Number  of  Brokers 
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7.2  End  to  End  Single  Clause  Latency 

The  input  to  output  latency  of  the  100X  JBI  was  detennined  from  the  amount  of 
time  necessary  to  process  an  information  object.  Of  interest  for  this  experiment  was  the 
determination  of  the  time  required  to  completely  process  an  incoming  information  object 
with  a  predicate  that  consisted  of  a  single  clause  on  one  processor.  The  number  of 
predicates  (subscribers)  was  increased  for  iteration  during  the  experiment,  and  the  time  to 
completely  process  each  iteration  was  measured. 

7.2.1  Experiment 

In  Table  VI  is  presented  the  experimental  parameters  for  the  determination  of  the 
end  to  end  latency  for  a  single  clause  for  the  100X  JBI  on  a  single  processor.  The 
specific  100X  JBI  architecture  for  this  experiment  is  presented  in  Figure  14.  This 
experiment  was  conducted  on  the  Coyote  Computer,  and  both  the  100X  JBI 
Implementation  and  the  JBI  1.2  Reference  Implementation  were  run  on  one  processor, 
respectively.  The  size  of  the  information  objects  was  1.3  kbytes. 


Table  VI.  End  to  End  Latency  Parameters 


Computer 

Coyote 

Processor  speed 

3.06  GHz 

Publication  Size 

1.3  kbytes 

Processors 

1 

Subscribers 

1  to  184 

39 


Figure  14.  100X  JBI  Architecture  for  end  to  end  latency  per  predicate  with  a  single  processor 


This  experiment  was  set  up  so  that  one  client  published  a  single  infonnation 
object  to  the  JBI  server.  The  number  of  subscribers  was  incrementally  increased  from  4 
to  184.  The  same  clause,  the  trivial  XPATH  expression  (/metadata/info/size>0),  was 
used  by  each  of  the  predicates  (subscribers).  When  the  Broker  received  the  information 
object,  it  evaluated  the  clause  and  determined  which  subscribers  requested  the 
information  object.  Since  all  of  the  subscribers  had  requested  it,  the  information  object 
was  then  disseminated  to  all  the  subscribers.  The  end  to  end  latency  for  this  experiment 
was  the  time  from  when  the  information  object  was  first  submitted  to  the  publisher 
catcher  to  when  the  last  information  object  was  disseminated  to  the  last  subscriber. 


7.2.2  Results 

In  Figure  15  is  presented  a  plot  of  the  detennined  latency  (ms)  verses  the  number 
of  predicates  (subscribers)  for  both  the  1.2  JBI  Reference  Implementation  and  the  100X 
JBI.  Linear  regression  analyses  have  been  performed  on  the  data,  and  the  calculated  best 
fitting  straight  lines  are  presented  in  the  figure.  The  slope  of  the  linear  least  squares  fit 
represented  the  increase  in  end  to  end  latency  per  time  unit  incurred  for  additional 
subscribers.  Given  this,  the  data  showed  that  each  subscriber  added  about  2.1  ms  of 
latency,  on  average,  to  the  system  for  the  JBI  1.2  Reference  Implementation  and  100  ps 
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for  the  100X  JBI.  The  100X  JBI  resulted  in  a  21  times  improvement  in  incurred  latency 
when  compared  with  the  JBI  1.2  Reference  Implementation.  With  184  subscribers  the 
latency  was  420  ms  for  the  JBI  1.2  Reference  Implementation,  and  19  ms  for  the  100X 
JBI  implementation.  Because  both  of  these  implementations  were  run  on  a  single 
processor,  the  improvement  in  the  latency  of  the  100X  JBI  implementation  was  attributed 
mainly  to  the  faster  execution  of  the  C  and  C++  codes  of  the  100X  v.  the  speed  of  JAVA 
executions  of  the  Reference  Implementation. 


JB1 1 .2,0.1  VS  100X  JBI 


0  2®  4®  6©  §©  10®  12®  14®  1@®  1§0  20® 


♦  JBI  1. 2.0.1  (ms) 

■  100X  JBI-2  Sub  Nodes  (ms) 

- Linear  (JBI  1. 2.0.1) 

- Linear  (100X  JBI) 


Network:  Gigabit 
Ethernet 


Publishers;  1 
Metadata  Size:;  1,3k 

HitRate:  100% 


Predicate  -  Tfl^Ji: 
(/rnetadiata/irrfd/size^®)) 


Number  of  Subscribers 


Figure  15.  End  to  end  latency  v.  the  number  of  subscribers  (predicate  clauses) 
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7.3  Predicate  Complexity 


While  the  previous  experiment  focused  on  detennining  the  increase  in  latency 
observed  by  adding  more  subscribers  (predicates),  the  following  experiment  focused  on 
the  detennination  of  how  the  complexity  of  each  subscriber's  XPATH  expression  affected 
the  100X  JBI  implementation  latency. 

7.3.1  Experiment 

By  definition,  each  XPATH  expression  is  a  predicate,  each  predicate  is  a 
conjunction  of  clauses,  and  each  clause  is  a  comparison  test  between  an  element  or  an 
attribute  of  the  XML  metadata  and  a  value.  For  example,  the  simple  predicate  used  in  the 
end-to-end  latency  test  (/metadata/info/size>0)  consisted  of  only  one  clause.  An  example 
of  a  predicate  with  two  clauses  is  (/metadata/info/size>0  and  /metadata/info/size<2000). 

For  this  experiment  the  predicate  complexity  was  defined  as  a  variable  of  the 
number  of  clauses  within  each  predicate.  The  experimental  parameters  are  presented  in 
Table  VII.  This  experiment  was  configured  with  a  single  publisher  publishing  1.3  kbyte 
information  objects,  and  with  ten  subscribers.  In  this  experiment  the  complexity  of  the 
predicates  was  changed.  The  architecture  of  this  experiment  is  presented  in  Figure  16. 


Table  VII.  Broker  Latency  Complexity  Parameters 


Computer 

Coyote 

Processor  speed 

3.06  GHz 

Publication  Size 

1.3  kbytes 

Publishing  Processors 

1 

Broker  Processors 

1 

Number  of  Subscribers 

10 
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Figure  16.  Architecture  for  end  to  end  latency  per  clause  with  a  single  processor. 

Short  circuiting  is  defined  as  when  a  system  only  evaluates  as  many  clauses  as 
necessary  to  arrive  at  a  final  answer.  For  example,  if  an  "or"  was  used  as  the  conjunction 
for  two  clauses  and  the  first  clause  resulted  in  a  true,  it  is  unnecessary  to  evaluate  the 
second  clause.  Short  circuiting  would  be  deleterious  for  this  experiment,  and  so  only  the 
final  comparison  was  true.  Because  short  circuiting  was  prevented,  this  experiment 
isolated  the  latency  effects  of  one  broker  evaluating  XML  documents. 
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7.3.2  Results 


In  Figure  17  is  presented  a  plot  of  the  determined  latency  verses  the  number  of 
clauses  per  subscriber.  A  linear  regression  has  been  performed  on  these  results,  and  the 
calculated  best  fitting  straight  line  is  also  presented  in  the  figure.  From  the  slope  of  the 
line,  a  10  clauses  increase,  (1  clause  per  subscriber  for  10  subscribers)  resulted  in  41  ps 
of  additional  latency  for  the  100X  JBI  implementation. 


0  50  100  150  200  250  300  350 

#  Clauses  per  Subscriber 


Figure  17.  Predicate  Complexity  vs.  End  to  End  Latency 
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7.4  100XJBI  Throughput 


Previous  experiments  determined  the  publisher  catcher's  Infonnation  Object 
handling  rate,  which  was  26  Mbytes  per  second  on  the  HHPC.  In  addition,  to  ensure  that 
the  disseminator  was  not  the  bottleneck,  the  predicates  used  guaranteed  that  a  specific 
information  object  only  met  one  of  the  subscriber's  requirements.  Therefore,  only  one 
Information  Object  needed  to  be  delivered  by  the  disseminator. 


7.4.1  Experiment 

Since  the  publisher  catcher  and  the  disseminators  were  assured  to  handle  the 
requirements  of  the  test,  the  1  publisher  catcher,  n  broker  and  1  disseminator 
configuration  of  the  100X  JBI  server  was  used  for  this  experiment  (Figure  18),  where  n  is 
the  number  of  brokers.  With  this  configuration,  the  pub  catchers  and  disseminators  were 
placed  on  the  same  node  and,  except  for  the  single  broker  configuration;  all  of  the  brokers 
were  placed  on  other  nodes,  such  that  each  processor  in  a  node  hosted  one  broker.  The 
actual  number  of  nodes  used  is  shown  in  Table  VIII.  The  number  of  subscribers  was  set 
at  300,  and  each  subscriber  had  a  two  clause  predicate,  which  resulted  in  a  600  clause 
brokering  job.  The  number  of  clients  (publishers)  varied  depending  on  the  desired 
publication  rate.  In  Table  IX  is  presented  the  experimental  parameters  to  determine  the 
maximum  throughput  rate. 


45 


Table  VIII.  Throughput  Parameters 


Computer 

Coyote 

Processor  speed 

3.06  GHz 

Predicates 

300 

Clauses/predicate 

600 

Publishing  Processors 

4-6 

Publisher  Catcher  Processors 

1 

Disseminator  Processors 

1 

Broker  Processors 

1  to  16 

Subscribers 

300 

Table  IX.  Throughput  Experimental  Conditions 


Run 

broker 

processors 

pub  catcher 
processors 

disseminator 

processors 

nodes 

1 

1 

1 

1 

1 

2 

2 

1 

1 

2 

3 

4 

1 

1 

3 

4 

8 

1 

1 

5 

5 

16 

1 

1 

9 
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Figure  18.  Architecture  for  Throughput  of  100X  JBI. 
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7.4.2  Results 


In  Figure  19  are  presented  the  results  of  the  100X  JBI  throughput  experiments. 
The  maximum  throughput  rates  achieved  were  534,  1033,  2032,  3939,  and  7036 
information  objects  per  second  for  1,  2,  4,  8  and  16  brokers,  respectively. 

In  Figure  20  is  presented  the  maximum  number  of  information  objects  that  were 
processed  in  one  second  and  the  optimum  number  of  information  objects  that  could  be 
processed  per  second  verses  the  number  of  brokers  used.  The  100X  JBI  implementation 
was  at  92%  optimum  for  the  8  processor  broker  configuration,  and  83%  optimum  with  16 
brokers. 


Throughput  Chart 


300  subscribers 
Predicate  ensures  that 
each  publication  is  delivered 
to  exactly  one  subscriber 
.3%  hit  ratio 

4-6  Publishers  w  ere  used 


Brokers 


0  2000  4000  6000  8000 

Publish  Rate  (io/s) 


10000 


Figure  19.  100X  JBI  Throughput  Results 
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Scalability 
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Figure  20.  100X  JBI  scalability 
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Figure  21.  Speedup  of  the  100X  JBI  verses  the  JBI  1.2  Reference  Implementation 
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When  compared  to  the  original  JBI  Reference  Implementation  1.2  (Figure  21), 
there  was  a  total  speed  up  of  347  times  in  terms  of  a  single  broker  for  the  100X  JBI 
Implementation,  and  4578  times  with  the  16  broker  configuration.  The  speedup  of  the 
one  broker  system  came  from  a  combination  of  rewriting  parts  of  the  code  that  were 
originally  written  in  JAVA  in  C  and  C++,  and  from  other  code  accelerations.  Additional 
acceleration  was  achieved  by  the  parallelizing  the  code  such  that  several  processors  were 
simultaneously  implementing  the  code. 
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8.  Summary 

The  results  of  the  experiments  designed  to  evaluate  the  core  service  speedups  of 
the  100X  JBI  architecture  have  shown  speedups  of  up  to  4578  times  verses  the  JBI 
reference  implementations.  Further  experiments  are  ongoing  to  evaluate  and  demonstrate 
the  100X  JBI  architecture,  and  will  be  presented  in  the  final  report. 
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APPENDIX  Symbols,  Abbreviations  and  Acronyms 


100X 

One  hundred  times  speedup 

AFRL 

Air  Force  Research  Laboratory 

C++ 

A  general  purpose  computer  programming  language.  Originally 
known  as  C  with  Classes 

CAPI 

Common  Application  Programming  Interface 

http 

HyperText  Transfer  Protocol 

IOR 

Information  Object  Repository 

IMS 

Information  Management  Staff 

IP 

Internet  Protocol 

JBI 

Joint  Battlespace  Infosphere 

JMS 

Java  Messaging  Services 

JNI 

Java™  Native  Interface  is  a  standard  programming  interface  for 
writing  Java  native  methods  and  embedding  the  Java™  virtual 
machine  into  native  applications.  The  primary  goal  is  binary 
compatibility  of  native  method  libraries  across  all  Java  virtual 
machine  implementations  on  a  given  platform. 

JniConnection 

JniConnection  is  the  Java  Native  Interface  (JNI)  wrapper  to  call 
our  C++  JBI  client  libraries  from  java 

JniConnectionService 

The  Java  Native  Interface  C++  code  that  enables  100X  JBI 
clients  in  C++  or  Java  to  communicate  to  the  original  JBI  Java 

server  code  that  provides  the  security  framework. 

MDR 

Metadata  Repository 

MPI 

Message  Passing  Interface 

POST 

Submits  user  data  (e.g.  from  a  HTML  form)  to  the  identified 
resource.  The  data  is  included  in  the  body  of  the  request. 

RBAC 

Role  Based  Access  Control 

SSH 

Secure  Shell  (protocol) 

SSL 

Secure  Socket  Layer 

VPN 

Virtual  Private  Network 
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XML 

Extensible  Markup  Language 

XMP 

Extensible  Metadata  Platform 

XPATH 

XMP  Path  Language 
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